{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:43.414025Z",
     "start_time": "2020-05-15T08:09:43.410933Z"
    }
   },
   "outputs": [],
   "source": [
    "# Data Processing\n",
    "import pandas as pd\n",
    "import re\n",
    "import sys\n",
    "# BERT\n",
    "import torch\n",
    "from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM\n",
    "from scipy.spatial.distance import cosine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:45.072678Z",
     "start_time": "2020-05-15T08:09:45.053013Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_pickle(\"../data/preprocessed_dataset\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:45.145796Z",
     "start_time": "2020-05-15T08:09:45.135336Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>questionTitle</th>\n",
       "      <th>questionText</th>\n",
       "      <th>answerText</th>\n",
       "      <th>reflection</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How can I keep a long distance relationship going?</td>\n",
       "      <td>How can I keep a long distance relationship going? We weren't long distance until he joined the military. I love him and I know he loves, me but it's complicated. He said he's not going to find someone else, but I'm afraid. How do I keep our relationship going?</td>\n",
       "      <td>Hello . You are asking a very good question about how to sustain a long - distance relationship . Although maintaining a long - distance relationship has its challenges , with the proper communication , commitment , and understanding , many long - distance couples are able to thrive and maintain a close connection . Without knowing more about the complicated nature of your relationship , I wonder whether your boyfriend has given you reason not to trust him that makes afraid that he will find someone else . Has he cheated on you in the past and has shown romantic interest to another person that made you fear losing him ? Or , is your thought just a fear but not based on evidence ? Knowing that distinction is important because if it is the latter , you may benefit from refocusing on the wonderful qualities about your partner that makes you feel good about him and the relationship rather than focusing on the unknown or uncertainty of the future . The more you focus on what if situations , the more you may feel anxious about a reality that is not accurate and make you act in ways that are insecure . However , if there is reason for you to question his fidelity , you may have to speak to your boyfriend about how to build trust in the context of a long - distance relationship . To help the conversation , you may need to consider what you may need to experience or receive as support to feel safe in the relationship to build trust . Is that you wish him to contact you regularly , or to include you more in his life , or to make a clear commitment ? For many of my clients in my private practice , that may include talking to their partner often and using a variety of modalities including text , phone , and Skype . Its hard to believe in a relationship when you never talk to your partner , and its hard to build a relationship when you dont know whats going on in your partners life . Other times , it is Making sure they talk often to their long - distance partner so that they can participate in each others lives and to feel their presence . . Regular communication , understanding and caring is the key to sustaining any relationship , but this is especially true for long - distance ones . Dr . Virginia Chowwww . PsychologyResource . ca</td>\n",
       "      <td>You are asking a very good question about how to sustain a long - distance relationship</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>How can I ask my boyfriend about who he's texting?</td>\n",
       "      <td>How can I ask my boyfriend about who he's texting? We've been in a long distance relationship for two and a half years. I recently saw his phone and saw the people he texts the most and one of them was a female coworker. I don't know how to approach this situation. How do I ask him about it?.</td>\n",
       "      <td>I agree with Sherry that in a close intimate relationship , you are entitled to ask questions about his relationship with significant others . These questions help couples to build connection and trust . Its based on the idea that if you reach out to him for whatever reason ( support , openess , understanding , empathy ) , you can count on him and can expect him to be responsive . How he responds to your question will give you an idea whether he helps you to feel more emotionally secure and builds trust or if you feel that you cannot be open with him . If your partner responds in an open and understanding manner , it usually indicates that he cares about your feelings and values your importance . If he responds in a defensive manner , it could mean that he does not like that you are questioning your trust in him or that he has something to hide . Either way , you may wish to explain that building trust is something that is very important to you in a relationship and that talking to him openly helps to foster that . If he continues to be defensive or evasive , then there might be some bigger issues at stake and the two of you may benefit from couples counselling or having a discussion about the values that are important to you in the relationship and how the two of you will go about supporting those values with actions . Dr . Virginia Chowwww . PsychologyResource . ca</td>\n",
       "      <td>These questions help couples to build connection and trust</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Why do I feel sad all the time?</td>\n",
       "      <td>Why do I feel sad all the time? I just feel sad all the time and I don't like anyone in my family. I feel like they're trying to control me and won't let me grow.</td>\n",
       "      <td>Hello , While one can be sad from time to time , feeling sad all the time could be a sign of depression . If you feel sad on most days , it is worthwhile speaking to a psychologist to determine whether you suffer clinical depression . Feeling sadness is a normal response to loss , whether you lose a family , friend , job , or something you deem important in your life . However , feeling sadness all the time is a signal that you are not happy about something in your life . If it is related to your family and often relationship with family members could affect our self - esteem and self - worth , then perhaps it is worthwhile exploring what you find unhappy about your relationship with your family . Is it that you do not feel heard , supported or loved ? Is it that you feel disrespected , disregarded or feel unimportant ? These are just a few examples of common issues that people report that contribute to their depression . To help you to uncover your reasons for your sadness or depression and to cope with them in a healthier and more constructive way , it may be helpful to contact a therapist who could support you with this process . You do not have to deal with it alone . Dr . Virginia Chow , Montreal Psychologist . For more information about depression , please consult my website at www . PsychologyResource . ca</td>\n",
       "      <td>If you feel sad on most days , it is worthwhile speaking to a psychologist to determine whether you suffer clinical depression</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The Underlying Causes of Being Possessive</td>\n",
       "      <td>The Underlying Causes of Being Possessive I am extremely possessive in my relationships and this is hurting my friendships. How can I fix my underlying issues?</td>\n",
       "      <td>Hi there . Its great you are able to realize there are other issues going on with someone who feels possessive . At the root of it is fear . Fear of losing someone , fear of being alone , fear of not being good enough . All those fears can lead to low self - esteem and feeling like you have to control other people so you dont lose them . The thing is , controlling other people only pushes them away . Vicious circle , right ? What I would suggest is some cognitive therapy to change those underlying ways of thinking . You can start with this assignment . Write down all the things you think about yourself , positive and not - so - positive . Then rewrite those not - so - positive things so they are positive . For example , thinking something like , Im too pushy , can be rewritten as , Im assertive and I go after what I want . It can be hard to do since we tend to get stuck in our negative ways of thinking about ourselves . If you have someone you trust , you can ask for their help as well since most likely they see you differently than you see yourself . Finding a good cognitive therapist can help you further , but if thats not an option for you right now , there are lots of self - help books and websites that are out there . Youve already taken the first step , so keep moving forward .</td>\n",
       "      <td>Its great you are able to realize there are other issues going on with someone who feels possessive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I get depressed whenever anyone I care about is too busy to be with me. What do I do?</td>\n",
       "      <td>I get depressed whenever anyone I care about is too busy to be with me. What do I do? My last relationships have ended horribly. They just up and abandoned me. One of them I have never gotten closure with over it, leaving me emotionally wrecked. I know something's wrong with me but I don't know how to even start trying to better myself.\\nI get into these depressed like states whenever anyone I care about is too busy to be with me. I know that I shouldn't act like that because people are genuinely busy, but I feel like they're doing it just because they don't want to be around me.</td>\n",
       "      <td>Frequently when working with clients , I utilize the A - B - C model developed by Albert Ellis , the founder of Rational Emotive Behavior Theory . The theory basically states that you can not control an event but you can control your reaction to the emotions associated with said event . For instance here is an example : If you look at the model , B has a direct effect on C . The thought is if you change your reaction to A , then C will have a better consequence , such as : This is just an example but if you actively work to replace the negative thought patterns that effect your behavior , studies show that your overall outlook and mood will improve . If this model resonates with you , I recommend Feeling better , getting better , staying better by Alber Ellis . In addition , looking for support groups in your area are a great source of support when you are feeling abandoned or alone .</td>\n",
       "      <td>Frequently when working with clients , I utilize the A - B - C model developed by Albert Ellis , the founder of Rational Emotive Behavior Theory</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                           questionTitle  \\\n",
       "0  How can I keep a long distance relationship going?                                      \n",
       "1  How can I ask my boyfriend about who he's texting?                                      \n",
       "2  Why do I feel sad all the time?                                                         \n",
       "3  The Underlying Causes of Being Possessive                                               \n",
       "4  I get depressed whenever anyone I care about is too busy to be with me. What do I do?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 questionText  \\\n",
       "0  How can I keep a long distance relationship going? We weren't long distance until he joined the military. I love him and I know he loves, me but it's complicated. He said he's not going to find someone else, but I'm afraid. How do I keep our relationship going?                                                                                                                                                                                                                                                                                                                                        \n",
       "1  How can I ask my boyfriend about who he's texting? We've been in a long distance relationship for two and a half years. I recently saw his phone and saw the people he texts the most and one of them was a female coworker. I don't know how to approach this situation. How do I ask him about it?.                                                                                                                                                                                                                                                                                                        \n",
       "2  Why do I feel sad all the time? I just feel sad all the time and I don't like anyone in my family. I feel like they're trying to control me and won't let me grow.                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "3  The Underlying Causes of Being Possessive I am extremely possessive in my relationships and this is hurting my friendships. How can I fix my underlying issues?                                                                                                                                                                                                                                                                                                                                                                                                                                              \n",
       "4  I get depressed whenever anyone I care about is too busy to be with me. What do I do? My last relationships have ended horribly. They just up and abandoned me. One of them I have never gotten closure with over it, leaving me emotionally wrecked. I know something's wrong with me but I don't know how to even start trying to better myself.\\nI get into these depressed like states whenever anyone I care about is too busy to be with me. I know that I shouldn't act like that because people are genuinely busy, but I feel like they're doing it just because they don't want to be around me.   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              answerText  \\\n",
       "0  Hello . You are asking a very good question about how to sustain a long - distance relationship . Although maintaining a long - distance relationship has its challenges , with the proper communication , commitment , and understanding , many long - distance couples are able to thrive and maintain a close connection . Without knowing more about the complicated nature of your relationship , I wonder whether your boyfriend has given you reason not to trust him that makes afraid that he will find someone else . Has he cheated on you in the past and has shown romantic interest to another person that made you fear losing him ? Or , is your thought just a fear but not based on evidence ? Knowing that distinction is important because if it is the latter , you may benefit from refocusing on the wonderful qualities about your partner that makes you feel good about him and the relationship rather than focusing on the unknown or uncertainty of the future . The more you focus on what if situations , the more you may feel anxious about a reality that is not accurate and make you act in ways that are insecure . However , if there is reason for you to question his fidelity , you may have to speak to your boyfriend about how to build trust in the context of a long - distance relationship . To help the conversation , you may need to consider what you may need to experience or receive as support to feel safe in the relationship to build trust . Is that you wish him to contact you regularly , or to include you more in his life , or to make a clear commitment ? For many of my clients in my private practice , that may include talking to their partner often and using a variety of modalities including text , phone , and Skype . Its hard to believe in a relationship when you never talk to your partner , and its hard to build a relationship when you dont know whats going on in your partners life . Other times , it is Making sure they talk often to their long - distance partner so that they can participate in each others lives and to feel their presence . . Regular communication , understanding and caring is the key to sustaining any relationship , but this is especially true for long - distance ones . Dr . Virginia Chowwww . PsychologyResource . ca   \n",
       "1  I agree with Sherry that in a close intimate relationship , you are entitled to ask questions about his relationship with significant others . These questions help couples to build connection and trust . Its based on the idea that if you reach out to him for whatever reason ( support , openess , understanding , empathy ) , you can count on him and can expect him to be responsive . How he responds to your question will give you an idea whether he helps you to feel more emotionally secure and builds trust or if you feel that you cannot be open with him . If your partner responds in an open and understanding manner , it usually indicates that he cares about your feelings and values your importance . If he responds in a defensive manner , it could mean that he does not like that you are questioning your trust in him or that he has something to hide . Either way , you may wish to explain that building trust is something that is very important to you in a relationship and that talking to him openly helps to foster that . If he continues to be defensive or evasive , then there might be some bigger issues at stake and the two of you may benefit from couples counselling or having a discussion about the values that are important to you in the relationship and how the two of you will go about supporting those values with actions . Dr . Virginia Chowwww . PsychologyResource . ca                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "2  Hello , While one can be sad from time to time , feeling sad all the time could be a sign of depression . If you feel sad on most days , it is worthwhile speaking to a psychologist to determine whether you suffer clinical depression . Feeling sadness is a normal response to loss , whether you lose a family , friend , job , or something you deem important in your life . However , feeling sadness all the time is a signal that you are not happy about something in your life . If it is related to your family and often relationship with family members could affect our self - esteem and self - worth , then perhaps it is worthwhile exploring what you find unhappy about your relationship with your family . Is it that you do not feel heard , supported or loved ? Is it that you feel disrespected , disregarded or feel unimportant ? These are just a few examples of common issues that people report that contribute to their depression . To help you to uncover your reasons for your sadness or depression and to cope with them in a healthier and more constructive way , it may be helpful to contact a therapist who could support you with this process . You do not have to deal with it alone . Dr . Virginia Chow , Montreal Psychologist . For more information about depression , please consult my website at www . PsychologyResource . ca                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "3  Hi there . Its great you are able to realize there are other issues going on with someone who feels possessive . At the root of it is fear . Fear of losing someone , fear of being alone , fear of not being good enough . All those fears can lead to low self - esteem and feeling like you have to control other people so you dont lose them . The thing is , controlling other people only pushes them away . Vicious circle , right ? What I would suggest is some cognitive therapy to change those underlying ways of thinking . You can start with this assignment . Write down all the things you think about yourself , positive and not - so - positive . Then rewrite those not - so - positive things so they are positive . For example , thinking something like , Im too pushy , can be rewritten as , Im assertive and I go after what I want . It can be hard to do since we tend to get stuck in our negative ways of thinking about ourselves . If you have someone you trust , you can ask for their help as well since most likely they see you differently than you see yourself . Finding a good cognitive therapist can help you further , but if thats not an option for you right now , there are lots of self - help books and websites that are out there . Youve already taken the first step , so keep moving forward .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
       "4  Frequently when working with clients , I utilize the A - B - C model developed by Albert Ellis , the founder of Rational Emotive Behavior Theory . The theory basically states that you can not control an event but you can control your reaction to the emotions associated with said event . For instance here is an example : If you look at the model , B has a direct effect on C . The thought is if you change your reaction to A , then C will have a better consequence , such as : This is just an example but if you actively work to replace the negative thought patterns that effect your behavior , studies show that your overall outlook and mood will improve . If this model resonates with you , I recommend Feeling better , getting better , staying better by Alber Ellis . In addition , looking for support groups in your area are a great source of support when you are feeling abandoned or alone .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       \n",
       "\n",
       "                                                                                                                                         reflection  \n",
       "0  You are asking a very good question about how to sustain a long - distance relationship                                                           \n",
       "1  These questions help couples to build connection and trust                                                                                        \n",
       "2  If you feel sad on most days , it is worthwhile speaking to a psychologist to determine whether you suffer clinical depression                    \n",
       "3  Its great you are able to realize there are other issues going on with someone who feels possessive                                               \n",
       "4  Frequently when working with clients , I utilize the A - B - C model developed by Albert Ellis , the founder of Rational Emotive Behavior Theory  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:45.221882Z",
     "start_time": "2020-05-15T08:09:45.218836Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2337, 4)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:45.305532Z",
     "start_time": "2020-05-15T08:09:45.301335Z"
    }
   },
   "outputs": [],
   "source": [
    "df.questionText = df.questionText.astype(str)\n",
    "df.questionTitle = df.questionTitle.astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:45.389843Z",
     "start_time": "2020-05-15T08:09:45.385902Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df[['questionTitle','questionText','reflection']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:45.550614Z",
     "start_time": "2020-05-15T08:09:45.468731Z"
    }
   },
   "outputs": [],
   "source": [
    "df[\"context_len\"] = df[\"questionText\"].str.len()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading BERT UNCASED BASE Pretrained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:45.645180Z",
     "start_time": "2020-05-15T08:09:45.640080Z"
    }
   },
   "outputs": [],
   "source": [
    "def initialize_tokenizer(pretrained_model):\n",
    "    # Load pre-trained model tokenizer (vocabulary)\n",
    "    tokenizer = BertTokenizer.from_pretrained(pretrained_model)\n",
    "    return tokenizer\n",
    "\n",
    "def get_index(tokenized_answer, tokenized_context):\n",
    "    result=[]\n",
    "    sll=len(tokenized_answer)\n",
    "    for ind in (i for i,e in enumerate(tokenized_context) if e==tokenized_answer[0]):\n",
    "        if tokenized_context[ind:ind+sll]==tokenized_answer:\n",
    "            result = [ind,ind+sll-1]\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:50.944811Z",
     "start_time": "2020-05-15T08:09:45.720296Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertModel(\n",
       "  (embeddings): BertEmbeddings(\n",
       "    (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "    (position_embeddings): Embedding(512, 768)\n",
       "    (token_type_embeddings): Embedding(2, 768)\n",
       "    (LayerNorm): BertLayerNorm()\n",
       "    (dropout): Dropout(p=0.1, inplace=False)\n",
       "  )\n",
       "  (encoder): BertEncoder(\n",
       "    (layer): ModuleList(\n",
       "      (0): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (1): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (2): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (3): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (4): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (5): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (6): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (7): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (8): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (9): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (10): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (11): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (pooler): BertPooler(\n",
       "    (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "    (activation): Tanh()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pretrained_model = 'bert-base-uncased'\n",
    "tokenizer = initialize_tokenizer(pretrained_model)\n",
    "model = BertModel.from_pretrained(pretrained_model)\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:50.953512Z",
     "start_time": "2020-05-15T08:09:50.946958Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_contextual_vector(model, tokenizer, question, context):\n",
    "    encoded_answer_context = \"[CLS] \" + context + \" [SEP]\"\n",
    "    #print (encoded_answer_context)\n",
    "        # maximum words that BERT can have is 512\n",
    "    encoded_answer_context = encoded_answer_context\n",
    "    tokenized_encoded_answer_context = tokenizer.tokenize(encoded_answer_context)[:512]\n",
    "\n",
    "    indexed_encoded_answer_context = tokenizer.convert_tokens_to_ids(tokenized_encoded_answer_context)\n",
    "    segments_ids_encoded_answer_context= [1] * len(tokenized_encoded_answer_context)\n",
    "    tokens_tensor_encoded_answer_context = torch.tensor([indexed_encoded_answer_context])\n",
    "    segments_tensors_encoded_answer_context = torch.tensor([segments_ids_encoded_answer_context])\n",
    "\n",
    "    # Predict hidden states features for each layer\n",
    "    with torch.no_grad():\n",
    "        encoded_layers_answer_context, _ = model(tokens_tensor_encoded_answer_context, segments_tensors_encoded_answer_context)\n",
    "    token_vecs_answer_context= encoded_layers_answer_context[11][0]\n",
    "\n",
    "    tokenized_answer = tokenizer.tokenize(question)\n",
    "    indexes = get_index(tokenized_answer, tokenized_encoded_answer_context)\n",
    "\n",
    "    if len(indexes) == 0:\n",
    "        print (\"check if question title is present in the question context\")\n",
    "        return None\n",
    "    else:\n",
    "        first_index, last_index = indexes[0], indexes[1] \n",
    "        return torch.mean(token_vecs_answer_context[first_index:last_index + 1], dim=0)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:09:50.965639Z",
     "start_time": "2020-05-15T08:09:50.956036Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>questionTitle</th>\n",
       "      <th>questionText</th>\n",
       "      <th>reflection</th>\n",
       "      <th>context_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How can I keep a long distance relationship going?</td>\n",
       "      <td>How can I keep a long distance relationship going? We weren't long distance until he joined the military. I love him and I know he loves, me but it's complicated. He said he's not going to find someone else, but I'm afraid. How do I keep our relationship going?</td>\n",
       "      <td>You are asking a very good question about how to sustain a long - distance relationship</td>\n",
       "      <td>261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>How can I ask my boyfriend about who he's texting?</td>\n",
       "      <td>How can I ask my boyfriend about who he's texting? We've been in a long distance relationship for two and a half years. I recently saw his phone and saw the people he texts the most and one of them was a female coworker. I don't know how to approach this situation. How do I ask him about it?.</td>\n",
       "      <td>These questions help couples to build connection and trust</td>\n",
       "      <td>293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Why do I feel sad all the time?</td>\n",
       "      <td>Why do I feel sad all the time? I just feel sad all the time and I don't like anyone in my family. I feel like they're trying to control me and won't let me grow.</td>\n",
       "      <td>If you feel sad on most days , it is worthwhile speaking to a psychologist to determine whether you suffer clinical depression</td>\n",
       "      <td>162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The Underlying Causes of Being Possessive</td>\n",
       "      <td>The Underlying Causes of Being Possessive I am extremely possessive in my relationships and this is hurting my friendships. How can I fix my underlying issues?</td>\n",
       "      <td>Its great you are able to realize there are other issues going on with someone who feels possessive</td>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I get depressed whenever anyone I care about is too busy to be with me. What do I do?</td>\n",
       "      <td>I get depressed whenever anyone I care about is too busy to be with me. What do I do? My last relationships have ended horribly. They just up and abandoned me. One of them I have never gotten closure with over it, leaving me emotionally wrecked. I know something's wrong with me but I don't know how to even start trying to better myself.\\nI get into these depressed like states whenever anyone I care about is too busy to be with me. I know that I shouldn't act like that because people are genuinely busy, but I feel like they're doing it just because they don't want to be around me.</td>\n",
       "      <td>Frequently when working with clients , I utilize the A - B - C model developed by Albert Ellis , the founder of Rational Emotive Behavior Theory</td>\n",
       "      <td>585</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                           questionTitle  \\\n",
       "0  How can I keep a long distance relationship going?                                      \n",
       "1  How can I ask my boyfriend about who he's texting?                                      \n",
       "2  Why do I feel sad all the time?                                                         \n",
       "3  The Underlying Causes of Being Possessive                                               \n",
       "4  I get depressed whenever anyone I care about is too busy to be with me. What do I do?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 questionText  \\\n",
       "0  How can I keep a long distance relationship going? We weren't long distance until he joined the military. I love him and I know he loves, me but it's complicated. He said he's not going to find someone else, but I'm afraid. How do I keep our relationship going?                                                                                                                                                                                                                                                                                                                                        \n",
       "1  How can I ask my boyfriend about who he's texting? We've been in a long distance relationship for two and a half years. I recently saw his phone and saw the people he texts the most and one of them was a female coworker. I don't know how to approach this situation. How do I ask him about it?.                                                                                                                                                                                                                                                                                                        \n",
       "2  Why do I feel sad all the time? I just feel sad all the time and I don't like anyone in my family. I feel like they're trying to control me and won't let me grow.                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "3  The Underlying Causes of Being Possessive I am extremely possessive in my relationships and this is hurting my friendships. How can I fix my underlying issues?                                                                                                                                                                                                                                                                                                                                                                                                                                              \n",
       "4  I get depressed whenever anyone I care about is too busy to be with me. What do I do? My last relationships have ended horribly. They just up and abandoned me. One of them I have never gotten closure with over it, leaving me emotionally wrecked. I know something's wrong with me but I don't know how to even start trying to better myself.\\nI get into these depressed like states whenever anyone I care about is too busy to be with me. I know that I shouldn't act like that because people are genuinely busy, but I feel like they're doing it just because they don't want to be around me.   \n",
       "\n",
       "                                                                                                                                         reflection  \\\n",
       "0  You are asking a very good question about how to sustain a long - distance relationship                                                            \n",
       "1  These questions help couples to build connection and trust                                                                                         \n",
       "2  If you feel sad on most days , it is worthwhile speaking to a psychologist to determine whether you suffer clinical depression                     \n",
       "3  Its great you are able to realize there are other issues going on with someone who feels possessive                                                \n",
       "4  Frequently when working with clients , I utilize the A - B - C model developed by Albert Ellis , the founder of Rational Emotive Behavior Theory   \n",
       "\n",
       "   context_len  \n",
       "0  261          \n",
       "1  293          \n",
       "2  162          \n",
       "3  159          \n",
       "4  585          "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_rows', None)\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.width', None)\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:21:53.917062Z",
     "start_time": "2020-05-15T08:09:50.968288Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "check if question title is present in the question context\n",
      "873\n",
      "check if question title is present in the question context\n",
      "880\n",
      "check if question title is present in the question context\n",
      "1291\n",
      "check if question title is present in the question context\n",
      "1327\n",
      "check if question title is present in the question context\n",
      "1346\n",
      "check if question title is present in the question context\n",
      "1429\n",
      "check if question title is present in the question context\n",
      "1482\n",
      "check if question title is present in the question context\n",
      "1595\n",
      "check if question title is present in the question context\n",
      "1764\n",
      "check if question title is present in the question context\n",
      "1772\n",
      "check if question title is present in the question context\n",
      "1786\n",
      "check if question title is present in the question context\n",
      "1798\n",
      "check if question title is present in the question context\n",
      "1843\n",
      "check if question title is present in the question context\n",
      "1897\n",
      "check if question title is present in the question context\n",
      "1927\n",
      "check if question title is present in the question context\n",
      "1935\n",
      "check if question title is present in the question context\n",
      "1941\n",
      "check if question title is present in the question context\n",
      "2014\n",
      "check if question title is present in the question context\n",
      "2018\n",
      "check if question title is present in the question context\n",
      "2022\n",
      "check if question title is present in the question context\n",
      "2026\n",
      "check if question title is present in the question context\n",
      "2051\n",
      "check if question title is present in the question context\n",
      "2063\n",
      "check if question title is present in the question context\n",
      "2076\n",
      "check if question title is present in the question context\n",
      "2112\n",
      "check if question title is present in the question context\n",
      "2198\n",
      "check if question title is present in the question context\n",
      "2212\n",
      "check if question title is present in the question context\n",
      "2221\n",
      "check if question title is present in the question context\n",
      "2225\n",
      "check if question title is present in the question context\n",
      "2325\n",
      "check if question title is present in the question context\n",
      "2333\n"
     ]
    }
   ],
   "source": [
    "# BERT only handles atmost 512 tokens\n",
    "contextual_vectors = []\n",
    "for i in range(df.shape[0]):\n",
    "    vec = get_contextual_vector(model, tokenizer, df.loc[i,\"questionTitle\"], df.loc[i,\"questionText\"])\n",
    "    if vec != None:\n",
    "        contextual_vectors.append(vec)\n",
    "    else:\n",
    "        print (i)\n",
    "        contextual_vectors.append([0 for i in range(768)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:21:53.931184Z",
     "start_time": "2020-05-15T08:21:53.919397Z"
    }
   },
   "outputs": [],
   "source": [
    "df[\"contextual_vector\"] = pd.Series(contextual_vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:21:53.956151Z",
     "start_time": "2020-05-15T08:21:53.935909Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5445990"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.memory_usage(deep=True, index = True).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save the Dataframe with Contextual Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:21:54.110830Z",
     "start_time": "2020-05-15T08:21:53.959381Z"
    }
   },
   "outputs": [],
   "source": [
    "df.to_pickle(\"../data/contextual_similarity_df\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load the Saved DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:21:54.265834Z",
     "start_time": "2020-05-15T08:21:54.114181Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_pickle(\"../data/contextual_similarity_df\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate Reflection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:24:52.249867Z",
     "start_time": "2020-05-15T08:24:52.247274Z"
    }
   },
   "outputs": [],
   "source": [
    "user_question = \"Why am I feeling uncomfortable at social situations, especially parties?\"\n",
    "\n",
    "user_question_context = \"I am an introvert and feel anxious in parties, for example I was at an event at USC last Friday, and after 20 min I left because I felt uncomfortable with all the new people there and I just wanted to be by myself. Why is that?\"\n",
    "\n",
    "user_question_context = user_question + user_question_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:24:52.685420Z",
     "start_time": "2020-05-15T08:24:52.585953Z"
    }
   },
   "outputs": [],
   "source": [
    "user_vec = get_contextual_vector(model, tokenizer, user_question, user_question_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:24:53.795495Z",
     "start_time": "2020-05-15T08:24:52.817152Z"
    }
   },
   "outputs": [],
   "source": [
    "for i in range(df.shape[0]):\n",
    "    df.loc[i, \"similarity_score\"] = 1 - cosine(user_vec, df.loc[i, \"contextual_vector\"]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:24:53.838117Z",
     "start_time": "2020-05-15T08:24:53.797406Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>questionTitle</th>\n",
       "      <th>questionText</th>\n",
       "      <th>reflection</th>\n",
       "      <th>context_len</th>\n",
       "      <th>contextual_vector</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>541</th>\n",
       "      <td>Why do I feel anxious and uncomfortable at parties or big social situations?</td>\n",
       "      <td>Why do I feel anxious and uncomfortable at parties or big social situations? Example would be homecoming? I was at my high school's home football game on Friday, and I left because I felt uncomfortable with all the people there and I just wanted to be by myself. Why is that?</td>\n",
       "      <td>Oh , that is so frustrating ! Have you ever heard of social anxiety ? Sounds like you may be suffering from it</td>\n",
       "      <td>275</td>\n",
       "      <td>[tensor(-0.2358), tensor(0.0309), tensor(0.1955), tensor(-0.0268), tensor(0.1350), tensor(-0.1140), tensor(-0.2105), tensor(0.7592), tensor(-0.5971), tensor(-0.2915), tensor(0.0340), tensor(-0.2916), tensor(-0.3366), tensor(0.1365), tensor(-0.2390), tensor(0.4136), tensor(0.8888), tensor(0.1472), tensor(-0.2646), tensor(0.7480), tensor(0.2853), tensor(-0.0635), tensor(-0.6850), tensor(0.5600), tensor(0.8233), tensor(0.0130), tensor(0.0011), tensor(-0.3409), tensor(-0.0205), tensor(-0.3522), tensor(0.3610), tensor(-0.0676), tensor(0.0142), tensor(0.3309), tensor(-0.2346), tensor(-0.2860), tensor(-0.1711), tensor(-0.1975), tensor(0.2608), tensor(0.1552), tensor(-0.4011), tensor(-0.9845), tensor(-0.8746), tensor(0.1349), tensor(-0.3269), tensor(-0.6830), tensor(0.5841), tensor(0.1373), tensor(0.0630), tensor(0.1155), tensor(-0.2734), tensor(0.4148), tensor(-0.4005), tensor(0.2638), tensor(0.0504), tensor(0.6712), tensor(0.0762), tensor(-0.5240), tensor(-0.4193), tensor(-0.6468), tensor(0.5736), tensor(-0.0652), tensor(0.0183), tensor(-0.4654), tensor(0.3044), tensor(0.2080), tensor(0.1291), tensor(-0.1683), tensor(-0.4428), tensor(-0.0117), tensor(-0.2723), tensor(-0.0311), tensor(-0.3912), tensor(-0.2502), tensor(-0.2450), tensor(0.5224), tensor(0.0274), tensor(-0.1409), tensor(-0.2258), tensor(0.2398), tensor(-0.2816), tensor(0.4019), tensor(-0.1102), tensor(0.6867), tensor(0.2167), tensor(0.0882), tensor(-0.0101), tensor(0.1982), tensor(0.2665), tensor(-0.1812), tensor(0.4052), tensor(0.0177), tensor(0.3830), tensor(0.2501), tensor(0.1050), tensor(-0.1326), tensor(0.5665), tensor(-0.4140), tensor(-0.2932), tensor(0.5466), ...]</td>\n",
       "      <td>0.918351</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    questionTitle  \\\n",
       "541  Why do I feel anxious and uncomfortable at parties or big social situations?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                            questionText  \\\n",
       "541  Why do I feel anxious and uncomfortable at parties or big social situations? Example would be homecoming? I was at my high school's home football game on Friday, and I left because I felt uncomfortable with all the people there and I just wanted to be by myself. Why is that?   \n",
       "\n",
       "                                                                                                         reflection  \\\n",
       "541  Oh , that is so frustrating ! Have you ever heard of social anxiety ? Sounds like you may be suffering from it   \n",
       "\n",
       "     context_len  \\\n",
       "541  275           \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          contextual_vector  \\\n",
       "541  [tensor(-0.2358), tensor(0.0309), tensor(0.1955), tensor(-0.0268), tensor(0.1350), tensor(-0.1140), tensor(-0.2105), tensor(0.7592), tensor(-0.5971), tensor(-0.2915), tensor(0.0340), tensor(-0.2916), tensor(-0.3366), tensor(0.1365), tensor(-0.2390), tensor(0.4136), tensor(0.8888), tensor(0.1472), tensor(-0.2646), tensor(0.7480), tensor(0.2853), tensor(-0.0635), tensor(-0.6850), tensor(0.5600), tensor(0.8233), tensor(0.0130), tensor(0.0011), tensor(-0.3409), tensor(-0.0205), tensor(-0.3522), tensor(0.3610), tensor(-0.0676), tensor(0.0142), tensor(0.3309), tensor(-0.2346), tensor(-0.2860), tensor(-0.1711), tensor(-0.1975), tensor(0.2608), tensor(0.1552), tensor(-0.4011), tensor(-0.9845), tensor(-0.8746), tensor(0.1349), tensor(-0.3269), tensor(-0.6830), tensor(0.5841), tensor(0.1373), tensor(0.0630), tensor(0.1155), tensor(-0.2734), tensor(0.4148), tensor(-0.4005), tensor(0.2638), tensor(0.0504), tensor(0.6712), tensor(0.0762), tensor(-0.5240), tensor(-0.4193), tensor(-0.6468), tensor(0.5736), tensor(-0.0652), tensor(0.0183), tensor(-0.4654), tensor(0.3044), tensor(0.2080), tensor(0.1291), tensor(-0.1683), tensor(-0.4428), tensor(-0.0117), tensor(-0.2723), tensor(-0.0311), tensor(-0.3912), tensor(-0.2502), tensor(-0.2450), tensor(0.5224), tensor(0.0274), tensor(-0.1409), tensor(-0.2258), tensor(0.2398), tensor(-0.2816), tensor(0.4019), tensor(-0.1102), tensor(0.6867), tensor(0.2167), tensor(0.0882), tensor(-0.0101), tensor(0.1982), tensor(0.2665), tensor(-0.1812), tensor(0.4052), tensor(0.0177), tensor(0.3830), tensor(0.2501), tensor(0.1050), tensor(-0.1326), tensor(0.5665), tensor(-0.4140), tensor(-0.2932), tensor(0.5466), ...]   \n",
       "\n",
       "     similarity_score  \n",
       "541  0.918351          "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.similarity_score == df.similarity_score.max()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-15T08:24:53.846260Z",
     "start_time": "2020-05-15T08:24:53.840847Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Youper: Oh , that is so frustrating ! Have you ever heard of social anxiety ? Sounds like you may be suffering from it\n"
     ]
    }
   ],
   "source": [
    "print (\"Youper:\", df[df.similarity_score == df.similarity_score.max()].reset_index().reflection[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
